wip(session-recordings): store session recordings to S3 #10142
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change does a number of things:
1. Adds a Kafka Connect cluster to dev, which includes the [Confluent
S3 Sink
Connector](https://docs.confluent.io/kafka-connect-s3-sink/current/overview.html).
This connector allows for pushing data to S3 from a Kafka Topic,
partitioned by some criteria. In this case we partition by
team_id,session_id,window_id. This should be scaleable although some
testing would be useful here.
2. Rather than retrieving the data from ClickHouse, we can get it
directly from S3.
3. Unpack and uncompress the nested rrweb event. Not really necessary
but a blob of base64 encoded gzipped data isn't very friendly.
4. Adds a script to create the Connector.
TODO: This doesn't:
1. add Kafka Connect to Helm
2. offload to MSK Connect deployment in production
3. remove the pushing of messages to ClickHouse. We still want a place
to push data to use to calculate the duration of the sessions.
|
This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the |
|
This PR was closed due to 2 weeks of inactivity. Feel free to reopen it if still relevant. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change does a number of things:
S3 Sink
Connector.
This connector allows for pushing data to S3 from a Kafka Topic,
partitioned by some criteria. In this case we partition by
team_id,session_id,window_id. This should be scaleable although some
testing would be useful here.
directly from S3.
but a blob of base64 encoded gzipped data isn't very friendly.
TODO: This doesn't:
to push data to use to calculate the duration of the sessions.
Refers to #2142
Problem
Changes
👉 Stay up-to-date with PostHog coding conventions for a smoother review.
How did you test this code?